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SUMMARY  AND  HIGHLIGHTS 


This  is  the  final  report  of  an  action  research  project  aimed  at 
developing  a prototype  method  for  increasing  the  benefits  derived 
from  training  courses  and  for  evaluating  the  contributions  of  such 
courses  to  management  development.  An  earlier  report^  analyzed 
information  obtained  just  before  and  immediately  after  training  from 
participants  in  the  Kepner-Tregoe  course  in  "Management  Problem 
Analysis  and  Decision  Making,"  and  from  their  supervisors.  The  pres- 
ent report  covers  the  results  of  a follow-up  conducted  eight  to  ten 
months  later. 

The  Concept 

Effective  training  means  a change  not  only  in  the  participant,  but  also 
in  his  supervisor,  in  higher  management,  and  in  the  total  work  situation. 

If  a course  is  to  be  effective,  management  must  be  ready  for  the  "changed 
man."  Sometimes  an  employee  is  unable  to  use  his  newly  acquired  knowledges 
and  skills  because  these  call  for  a change  in  established  ways  of  doing' 
things--changes  that  his  supervisors  and  associates  may  be  unable  or 
unwilling  to  accept. 

Thus,  any  particular  course  is  but  a phase  of  a continuing  process,  one 
step  in  a sequence  of  individual,  group,  and  organizational  changes. 
Training  evaluation,  therefore,  cannot  stop  with  the  trainee.  It  must 
also  take  into  account  the  role  played  by  the  supervisors  and  managers, 
as  well  as  the  interaction  among  all  parties  involved  in  planning  for 
training  and  for  its  application. 

The  Data 


Two  questionnaires  were  constructed  --  one  for  participants  and  the 
other  for  supervisors  (Appendix  A),  On  August  15,  1963,  these  question 
naires  were  mailed  to  the  54  participants  described  in  the  earlier 
report,  and  to  their  supervisors.  This  follow-up  covered  a period 
ranging  from  eight  to  ten  months  after  completion  of  the  course. 

Thirty-seven  participants  returned  completed  questionnaires  in  time  to 
be  included  in  the  analysis.  Twenty-five  supervisors  reported  on  36 
participants;  not  always  on  the  same  participants  who  reported  on 
themselves.  Information  from  both  the  participant  and  his  supervisor 
was  obtained  in  29  cases. 
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The  questions  dealt  solely  with  the  effectiveness  of  the  training  in  pro- 
ducing observable  changes  in  the  job  performance  of  the  participants. 
Neither  participants  nor  their  supervisors  were  asked  how  they  liked  the 
course,  what  they  thought  of  the  instructor,  how  good  the  presentations 
were,  etc.  Our  object  was  to  find  out  what  the  participants  actually 
did  with  the  training  after  they  had  received  it,  how  they  used  it  in 
their  work,  and  whether  the  amount  and  kinds  of  communication  and 
follow-up  that  took  place  between  participant  and  supervisor  made  a 
difference  in  the  participant's  application  of  his  newly  acquired 
learning. 

The  findings  pose  a number  of  questions  for  management.  Judgments 
must  be  made  as  to  whether  these  situations  are  acceptable: 

437o  of  the  participants  were  able  to  specify  in  detail 
how  they  had  applied  the  training  in  their  work; 

277o  were  unable  to  indicate  that  they  had  used  the  training 
at  all; 

In  33%  of  the  cases  there  was  no  discussion  of  the  training 
between  participants  and  their  supervisors  during  the  eight 
to  ten  months  following  completion  of  the  course; 

In  44%  of  the  cases  no  definite  follow-up  efforts  were  reported 
by  the  supervisors; 

In  56%  of  the  cases  supervisors  were  unable  to  specify  what,  if 
any,  use  the  participants  had  made  of  the  training; 

Feedback  on  the  first  phase  of  this  project  reached  36%  of  the 
supervisors  and  38%  of  the  participants. 

The  questions  raised  by  findings  like  these  become  more  insistent  in 
the  light  of  other  data  presented  in  this  report.  Thus,  the  evidence 
indicates  that§ 

i\sking  about  follow-up  plans  increases  the  likelihood  that 
follow-up  will  actually  occur; 

Feedback  promotes  follcw-up; 

Discussion  and  follow-up  efforts  by  the  supervisor  increase 
the  chances  that  the  participant  will  find  applications 
of  the  training  in  his  work. 

If  then,  the  training  situation  seems  unsatisfactory  at  present,  there 
appear  to  be  ways  of  improving  it. 
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Recommendation 


If  the  procedure  described  in  this  report  and  its  predecessor  is  used  as 
a *’one-shot”  affair,  then,  it  has  little  lasting  value.  To  realize 
the  potential  benefits,  it  is  necessary  that  the  procedure  be  carried 
out  methodically  and  on  a continuing  basis.  This  means,  among  other 
things,  insisting  upon  complete  return  of  questionnaires  from  both 
participants  and  their  supervisors.  If  ipanagement  treats  the  evalua- 
tion process  casually,  others  will  do  so  too.  The  result  can  be  an 
altogether  unwarranted  complacency  regarding  the  effectiveness  of  the 
entire  training  and  management  development  program. 

It  is  recommended  that  the  method  described  here,  or  appropriate 
variants  of  it,  be  applied  to  all  management  courses  on  a continuing 
basis. 


• -v  - 


Albert  S,  Glickman 
Chief 

Personnel  Research  Staff 
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I.  INTRODUCTION 


This  is  the  second  and  final  report  on  an  action  research  project 
aimed  at  developing  a prototype  method  for  increasing  the  benefits 
derived  from  training  courses  and  for  evaluating  the  contributions  of 
such  courses  to  management  development.  An  earlier  report  analyzed 
information  obtained  just  before  and  immediately  after  training  from 
participants  in  the  Kepner-Tregoe  course  in  Management  Problem  Analysis 
and  Decision  Making,  and  from  their  supervisors.'^  The  present  report 
covers  the  results  of  a follow-up  conducted  eight  to  ten  months  later. 

It  was  emphasized  in  the  first  report  that  effective  training  means 
change  not  only  in  the  participant,  but  also  in  his  supervisor,  in 
higher  management,  and  in  the  total  work  situation.  If  a training 
course  is  to  be  effective,  management  must  be  ready  for  the  "changed 
man."  Sometimes  an  employee  is  unable  to  use  his  newly  acquired  knowl- 
edges and  skills  because  these  call  for  changes  in  established  ways  of 
doing  things--changes  that  his  superiors  and  associates  may  be  unable  or 
unwilling  to  accept.  When  this  happens,  the  training  effort  is  wasted, 
if  not  actually  detrimental  to  the  participant  and  to  the  efficient 
functioning  of  the  organization. 

Within  this  broad  concept,  any  particular  course  is  but  a phase  of 
a continuing  training  process,  one  step  in  a sequence  of  individual, 
group,  and  organizational  changes.  Training  evaluation,  therefore,  can- 
not stop  with  the  trainee.  It  must  also  take  into  account  the  role  played 
by  supervisors  and  managers,  as  well  as  the  interaction  among  all  parties 
involved  in  planning  for  training  and  for  its  application. 


Harmon,  F.  L.  "A  Path  to  Management  Development  and  to  the  Measure- 
ment of  Its  Growth,"  U.  S.  Department  of  Agriculture,  Office  of  Personnel, 
Personnel  Research  Series  63-2 , 1963. 
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II.  WHAT  WE  LOOKED  FOR 


To  summarize  further  from  the  first  report,  five  general  questions 
define  the  evaluation  process,  as  follows: 

1,  What  specific  things  is  the  trainee  expected  to  ^ or  to  do 
differently  as  a result  of  this  training  experience? 

2o  How  did  the  individual  act  before  he  was  trained?  How  did  he 
act  after  training? 

3*  Are  the  long-term  effects  the  same  as  the  more  inmediate  changes 
in  behavior? 

4o  What,  precisely,  do  we  wish  to  evaluate? 

5.  What  use  is  to  be  made  of  the  evaluation  process? 

Besides  these  general  questions,  our  approach  was  determined  by  what 
we  considered  to  be  the  specific  features  of  a sound  training  program. 
Thus : 

1.  Both  participants  and  supervisors  should  display  some  advance 
knowledge  of  the  specific  objectives  of  the  training, 

2o  They  should  evidence  knowledge  of  the  employee’s  individual 
training  needs. 

3o  They  should  clearly  perceive  the  relation  of  this  particular 
training  to  the  participant’s  career  development. 

4,  Training  should  be  integrated  with  operations--supervisors  and 
participants  should  be  able  to  indicate  definite  ways  in  which 
the  employee  is  expected  to  use  the  training  in  his  work, 

5,  Communication  between  supervisor  and  employee  should  indicate 
that  both  are  involved  in  planning  the  training  effort. 

6,  Adequate  feedback  should  be  provided  to  the  supervisor,  as  well 
as  to  agency  and  departmental  training  staffs. 
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III.  WHAT  WE  DID 


Data  Collection 

Two  questionnaires  were  constructed--one  for  participants  and  the 
other  for  supervisors  (Appendix  A),  On  August  15,  1963,  the  Employee 
Development  Division  mailed  these  questionnaires  to  the  54  participants 
described  in  the  earlier  report,  and  to  their  supervisors.  Since  the 
Kepner-Tregoe  sessions  attended  by  these  participants  were  held  between 
October  15  and  December  14,  1962,  this  follow-up  covered  a period  ranging 
from  eight  to  ten  months  after  completion  of  the  course. 


Thirty-seven  participants,  or  approximately  697o  returned  completed 
questionnaires  in  time  to  be  included  in  the  analysis.  Twenty-five 
supervisors  reported  on  36  participants,  though  not  always  on  the  same 
participants  who  reported  on  themselves.  Information  from  both  the  par- 
ticipant and  his  supervisor  was  obtained  in  29  cases  or  547.  of  the 
original  group. ^ 


Data  Analysis 

The  questionnaire  data  were  scored  in  the  same  way  as  in  the  first 
report.  The  responses  to  each  question  were  content-analyzed,  then 
sorted  into  categories  according  to  the  quality  level  of  the  content. 

No  more  than  three  categories  were  used  in  any  case,  A value  of  ^ was 
always  given  to  the  ’’best'*  or  "preferred"  answer,  usually  involving  a 
report  of  specific  operational  gains  or  changes  attributed  to  the  train- 
ing. Zero  signified  no  report  of  change  or  benefit.  One  was  in  between. 
"Yes-No"  questions  were  scored  1_  or  0.  Details  and  examples  of  the 
scoring  of  each  question  are  given  in  Appendix  B. 


Since  not  all  questionnaires  were  returned,  it  is  necessary  to 
inquire  whether  those  who  did  respond  constitute  a representative  sample 
of  the  original  group.  A statistical  examination  of  the  data  supports 
the  assumption  of  representativeness.  (Appendix  C contains  the  detailed 
analyses. ) 


Continued  pressure  eventually  elicited  returns  from  8 more  parti- 
cipants and  6 more  supervisors  who  reported  on  7 participants.  None  of 
these  returns  could  be  included  in  the  data  of  this  report. 
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IV.  WHAT  WE  FOUND 


This  method  of  evaluation  was  designed  to  answer  certain  questions 
about  the  effectiveness  of  the  training  effort.  The  research  findings 
will  be  presented  in  terms  of  these  questions  according  to  the  following 
plan: 

1.  The  question  asked  of  the  data; 

2.  Relevant  questions  asked  of  participants  and  supervisors; 

3.  Analysis  of  questionnaire  responses; 

4.  Answer  provided  by  the  data  to  the  original  question; 

5.  Comments. 

Both  questionnaires  contained  five  main  items  including  "other 
comments."  These  items  covered  four  areas  of  inquiry: 

A.  Follow-up  by  supervisors; 

B.  Applications  of  the  training; 

C.  Feedback 

D.  Other  comments. 

The  results  will  be  presented  in  this  order. 
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A.  FOLLOW-UP  BY  SUPERVISORS 


Question  A.l:  How  much  and  what  kind  of  supervisor-part icipant  communi - 
cation  occurred  after  training? 

The  supervisors*  Follow-up  Question  1 asked:  HAS  THIS  EMPLOYEE 
DISCUSSED  THE  KEPNER-TREGOE  CONCEPTS  AND  METHODS  WITH  YOU  DURING 
THE  PAST  SIX  MONTHS  OR  SO? 

The  supervisors  reported  on  36  participants. 

677o  of  the  participants  had  discussed  the  training  with  their 
supervisors, 

337o  had  not  done  so. 

Corresponding  figures  for  a similar  question  asked  before  training 
were  557,  and  457o.  The  difference  between  these  "before*'  and  "after" 
percentages  is  not  statistically  significant. 

Follow-up  Question  2 asked:  a.  HAVE  YOU  BEEN  ABLE  TO  FOLLOW  UP  ON 
HIS  USE  OF  THE  KEPNER-TREGOE  TRAINING?  b.  IF  Y^,  IN  WHAT  SPECIFIC 
WAYS? 


The  distribution 
Level  2 
Level  1 
Level  0 


of  supervisors* 
25.07o 

30.570 

44.570 


responses  was: 


When  Level  2 and  Level  1 responses  are  combined,  supervisors  appear 
to  have  made  at  least  some  effort  to  follow-up  on  participants  slightly 
more  than  half  of  the  time.  Nearly  half  of  the  time,  however,  nothing 
along  this  line  has  been  done. 


The  question  exposed  to  management  judgment  by  these  findings  is: 
*'Is  the  extent  of  supervisory  follow-up  as  great  as  it  ought  to  be?" 
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Question  A„2s  How  consistent  were  supervisors  with  respect  to  follow-up 
discussions? 

There  were  26  participants  whose  supervisors  answered  both  the  pre- 
training and  the  follow-up  questions  about  discussions  of  the  course. 

The  distribution  of  their  responses  to  the  two  questions  is  shown  in 
Exhibit  1, 


Exhibit  1 


Supervisor' 

-Participant  Discussions 
and  After  the  Course 

Before 

Response 

Before 

Follow-up 

N 

Yes 

Yes 

10 

Yes 

No 

4 

No 

Yes 

6 

No 

No 

6 

26 

There  seems  to  have  been  little  consistency  in  the  pattern  of  dis- 
cussions on  the  part  of  individual  supervisor-participant  pairs.  The 
correlation  between  the, ’’before"  and  "after"  data  of  Exhibit  1 is  .35, 


a nonsignificant  value 
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Tetrachoric  r was  estimated  by  the  cosine  formula. 
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Question  A. 3:  How  reliable  are  expressed  intentions  about  follow-up? 

Immediately  after  the  course  supervisors  were  asked:  IN  WHAT  WAYS 
DO  YOU  PLAN  TO  FOLLOW  UP  ON  THIS  ExMPLOYEE^S  USE  OF  WHAT  HE  LEARNED  IN 
THE  COURSE?  (Question  Post-2) 

Let  us  see  to  what  extent  they  were  able  to  carry  out  their  plans. 
There  were  21  participants  whose  supervisors  responded  to  both  Question 
Post-2  and  Follow-up  2.  Exhibit  2 shows  the  distribution  of  these 
responses  by  levels. 


Exhibit  2 


Responses  by  Supervisors  of  21  Participants 
to  Questions  Post-2  and  Follow-up  2 


Follow-up  2 Level 
012  Total 

13 

7 

1 

Total  8 8 5 21 


The  exhibit  indicates  that  for  these  21  participants: 

(along  the  diagonal) 

in  7 cases,  supervisors  followed  up  as  specifically  as  they 
initially  planned  to  do; 

(above  the  diagonal) 

in  5 cases,  supervisors  did  not  implement  their  follow-up  plans 
as  specifically  as  they  initially  planned  to  do; 

in  8 cases,  supervisors  did  not  implement  their  follow-up  plans 
at  all; 

(below  the  diagonal) 

in  1 case,  a supervisor  who  had  not  initially  planned  to  follow 
up,  later  reported  some  follow-up  effort. 
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The  difference  between  intention  and  actual  achievement  of  the 
supervisors  is  highly  significant  statistically.'^ 

Immediately  following  training,  participants  were  asked:  WHAT  PLANS 
DO  YOU  HAVE  FOR  DISCUSSING  THIS  COURSE  WITH  YOUR  SUPERVISOR?  (Question  Post-  4) 

Twenty-five  participants  (out  of  the  37  in  our  follow-up  sample) 
responded  to  this  question,  and  all  25  indicated  some  plans  for  follow-up 
discussion  (Levels  1 and  2 combined).  Of  these  25: 

22  participants  did  carry  through  on  the  discussions,  according 
to  reports  of  their  supervisors; 

3 participants  did  not. 

Evidently,  participants  were  considerably  more  successful  than  super- 
visors in  carrying  out  their  intentions  in  regard  to  follow-up.  One  might 
wonder  why  this  should  be  so. 


A two-tailed  sign  test  was  significant  at  the  0.47o  level  of 
confidence. 
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Question  A<,4s  What  is  the  evidence  that  the  present  approach  promotes 
follow-up? 


In  the  first  report  it  was  mentioned  that  one  group  of  participants 
was  not  asked  Question  Post-4o  Eight  of  these  individuals  are  represented 
in  the  present  data. 

How  do  they  compare  with  the  28  participants  who  were  asked  Question 
Post-4  in  the  matter  of  follow-up  discussions  with  their  supervisors? 

23  of  the  28  participants  who  were  asked  Question  Post-4  engaged  in 
a subsequent  discussion; 

1 of  the  8 participants  who  were  not  asked  Question  Post-4  engaged 
in  a subsequent  discussiono 

The  difference  is  highly  significant  statistically,^ 

Evidently,  merely  asking  a participant  about  his  plans  tends  to 
promote  discussion  between  himself  and  his  supervisor. 

Thus  we  find  support  for  our  hypothesis  that  the  evaluation  procedure 
developed  in  this  project  can  lead  to  improvement  in  the  overall  training 
effort,  j_f  i_t  carried  out  consistently  and  on  a continuing  basis , 
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The  Fisher 


exact  probability  test 


yielded  a one-tailed  P of  0,267oo 
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APPLICATIONS  OF  THE  TRAINING 


Question  B.ls  Di_d  the  results  come  up  to  participants  * post-training 
expectations? 

Immediately  following  trairdng  the  participants  had  been  asked: 

GIVE  E)6\MPLES  of  SITUATIONS’  IN-'YOJR  OWN  WORK  IN’ WHICH  THE  CONCEPTS  AND  - 
METHODS  TAUGHT  IN  THIS  COURSE  WOULD  BE  APPLICABLE  ’ '(Qu©mibi\^  Post)- 2a)  The 
distribution  of  their  responses  at  that  time  was: 


Level  2 

46% 

Level  1 

40% 

Level  0 

14% 

Total 

100% 

Exhibit  3 shows  the  distributions  of  responses  for  the  29  partici- 
pants who  answered  both  Question  Post-2a  and  Follow-up  1 (which  asked 
about  actual  applications) » 


Exhibit  3 


Distributions  of  Participants’  Responses  to 
Questions  Post  2a  and  Follow  up  1 
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The  exhibit  indicates  that  of  29  participants: 

(along  the  diagonal) 

16 j including  3 who  did  not  expect  to  get  anything  out  of  the 
course  to  begin  with^  found  that  their  initial  expectations 
were  equalled^ 
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(above  the  diagonal) 

6 were  unable  to  document  useful  applications  to  the  extent 
that  they  had  expected; 

(below  the  diagonal) 

7 found  that  their  initial  expectations  were  surpassed. 

The  difference  between  the  two  distributions  is  not  significant. 
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Question  B.2o  How  many  participants  found  situations  in  which  the  training 
was  inapplicable?  What  were  the  reasons  given? 

Participant  Follow-up  Question  2 asked: 

a.  HAVE  THERE  BEEN  ANY  SITUATIONS  IN  WHICH  YOU  TRIED  TO  USE  THESE 
CONCEPTS  AND  METHODS,  BUT  THEY  DID  NOT  HELP  YOU  MUCH? 

b.  IF  Y^,  WHAT  DO  YOU  CONSIDER  TO  HAVE  BEEN  THE  REASONS?  GIVE 
SPECIFIC  EXAMPLES, 

14  of  the  37  participants,  or  387o  said:  Yes.  The  reasons  given  for 
inability  to  apply  the  Kepner-Tregoe  method  varied. 

6 said  that  their  decisions  are  predetermined  by  Departmental  policy, 
regulations,  legislation,  or  the  prior  decisions  of  others.  As 
one  expressed  it,  the  Kepner-Tregoe  method  *'does  not  lend  itself 
to  government  administrative  work,” 

3 blamed  their  lack  of  success  upon  the  fact  that  other  staff 
members  were  not  trained  in  the  Kepner-Tregoe. 

2 said  that  the  objectives  of  the  problems  which  they  tried  to  solve 
were  not  clearly  enough  defined. 

3 felt  that  they  lacked  sufficient  information  on  the  problem  in 
question,  although  one  of  these  pointed  out  that  his  failure  was 
rather  in  himself  than  in  the  method. 

Supervisors  were  not  specifically  asked  about  situations  in  which  a 
participant  tried  without  success  to  apply  the  Kepner-Tregoe  method. 
However,  the  Supervisor  Follow-up  Question  3c  did  provide  for  cases  in 
which  the  participant  was  reported  not  to  have  applied  his  training:  IF 
NO,  WHAT  DO  YOU  CONSIDER  TO  BE  THE  REASONS  HE  HAS  BEEN  UNABLE  TO  MAKE  USE 
OF  THESE  CONCEPTS  AND  METHODS? 

Only  four  supervisors  attempted  an  answer  to  this  question.  One  of 
these  agreed  with  the  three  participants  who  said  that  other  personnel  had 
not  been  trained  in  the  Kepner-Tregoe  approach.  The  other  ’’explanations” 
were  as  follows: 

”He  still  has  trouble  making  decisions.” 

”He  has  been  involved  in  completion  of  past  work  and  preparation 

for  transfer  to  a new  headquarters.” 

’’The  problems  he  faces  in  (his  research)  are  quite  unique  and 

difficult  of  solution  by  ordinary  methods,” 
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Question  B,3:  How  many  participants  were  able  to  use  the  training? 

Both  participants  and  supervisors  were  asked  about  actual  applica- 
tions of  the  training  in  the  participants*  daily  work.  The  questions  went 
as  follows; 

Participant  Follow-up  Question  la:  HAVE  THERE  BEEN  ANY  SITUATIONS  IN 
WHICH  THE  KEPNER-TREGOE  CONCEPTS  AND  METHODS  WERE  HELPFUL  TO  YOU  IN 
YOUR  WORK? 

b,  IF  GIVE  SPECIFIC  EXAMPLES. 

Supervisor  Follow-up  Question  3a:  HAVE  THERE  BEEN  ANY  SITUATIONS  IN 
WHICH  THESE  CONCEPTS  AND  METHODS  SEEM  TO  HAVE  BEEN  HELPFUL  TO  THIS 
EMPLOYEE  IN  HIS  WORK? 

bo  IF  Y^,  GIVE  SPECIFIC  EXAMPLES. 

Participants  and  supervisors  gave  these  responses: 


Participants 

Supervisors 

Level 

2 

437o 

227o 

Level 

1 

307. 

227o 

Level 

0 

277o 

567. 

A majority  of  the  participants  (Levels  1 and  2)  perceived  that  the 
course  had  been  helpful  to  them  in  their  work,  A majority  of  the  super- 
visors (Level  0)  did  not  perceive  that  the  course  had  been  helpful  to 
their  subordinates. 

Participant  Follow-up  Question  3 referred  to  use  of  the  Kepner-Tregoe 
training  in  a special  situation:  a.  HAVE  YOU  MADE  USE  OF  THESE 
CONCEPTS  AND  METHODS  IN  DISCUSSIONS  WITH  YOUR  SUPERVISOR  DURING  THE 
PAST  SIX  MONTHS  OR  SO? 

b.  IF  YES,  GIVE  EXAMPLES. 

The  percentages  of  participants  responding  to  this  question  at  each 
level  were: 


Level 

2 

307. 

Level 

1 

107. 

Level 

0 

607. 
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The  fact  that  the  majority  of  the  participants  did  not  make  use,  in 
discussions  with  their  supervisors,  of  the  concepts  and  methods  they  had 
learned  (Level  0),  offers  a possible  explanation  of  the  disparity  between 
the  two  groups  in  their  perception  of  the  helpfulness  of  the  course. 
Obviously,  it  would  be  difficult  for  a supervisor  to  know  if  the  concepts 
and  methods  were  helpful,  if  he  had  no  knowledge  of  or  exposure  to  them. 
The  next  analysis  pursues  this  point  further. 
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Question  B,4:  How  well  aware  of  these  appl i cations  are  the  supervi sors ? 

Responses  to  the  relevant  questions  were  obtained  from  both  the 
participant  (Follow-up  Question  1)  and  his  supervisor  (Follow-up  Question 
3)  in  29  cases.  Exhibit  4 shows  the  distributions  of  these  responses. 


Exhibit  4 

Responses  of  Participants  to  Follow-up  Question  1 
and  by  Their  Supervisors  to  Follow-up  Question  3 
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Total 

6 

6 

17 


Total  8 


7 14  29 


The  exhibit  indicates  for  29  participant-supervisor  pairs: 

(along  the  diagonal) 

11  pairs  agreed 
(above  the  diagonal) 

4 instances  in  which  supervisors  cited  more  application  than 
did  the  participants  themselves. 

(below  the  diagonal) 

14  instances  in  which  participants  cited  more  application 
than  did  their  supervisors,  including  10  in  which  the 
supervisors  could  cite  no  applications 

The  differences  betwgen  supervisor  and  participant  responses  are 
statistically  significant. 


A two-tailed  sign  test  was  significant  at  the  1.57o  level. 
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This  and  the  preceding  analysis  present  us  with  the  question:  *'Do 
we  have  adequate  communication  concerning  the  benefits  or  lack  of 
benefits  of  the  training  course?” 

Possibly,  communication  deficiencies  might  be  remedied  by  making 
provisions  for  supervisor  and  participant  to  review  together  the  ways 
in  which  the  training  can  be  used  at  work. 

The  next  analysis  may  throw  more  light  upon  this  question. 
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Question  Bo 5;  What  is  the  relationship  between  supervisor -participant 
communication  and  application  of  the  training? 

We  can  use  the  data  in  the  next  two  analyses  to  develop  an  answer. 

We  compared  the  participants  who  did  with  those  who  did  not  discuss 
the  training  with  their  supervisors,  with  respect  to  the  number  in  each 
group  who  found  applications  for  the  training  during  the  eight  to  ten 
months  following  completion  of  the  course.  A total  of  29  pairings  of 
supervisors  and  participants  were  involved. 

20  participant's*  supervisors  said  yes;  discussions  had  taken  place. 

17  of  these  participants  said  that  they  had  applied  the  training 
in  their  work  (Levels  1 and  2); 

3 of  the  participants  said  that  they  had  not  applied  the  training 
in  their  work  (Level  0). 

9 participants*  supervisors  said,  no;  discussions  had  not  taken 
place. 

4 of  these  participants  said  that  they  had  applied  the  training 
in  their  work; 

5 of  these  participants  said  that  they  had  not  applied  the  train- 
ing in  their  work. 

The  difference  between  the  two  groups  is  statistically  significant.^ 

We  made  a similar  comparison  of  participants  * responses  to  Follow-up 
Question  1 in  relation  to  their  supervisors  * responses  to  both  Follow-up 
Question  1 and  Follow-up  Question  2.  Again,  we  have  29  cases. 

12  participants*  supervisors  reported  both  discussions  and  specific 
follow-up  efforts  to  determine  what  use  these  employees  made  of 
the  training. 

11  of  these  participants  said  that  they  had  applied  the  training 
(Levels  1 and  2) ; 

1 of  these  participants  said  that  he  had  not  applied  the  train- 
ing (Level  0). 


^The  Fisher  exact  probability  test  yielded  a one-tailed  P of  47. 


17  participants*  supervisors  reported  omitting  one  or  both  follow-up 
stepso 

10  of  these  participants  said  that  they  had  applied  the  training; 

7 of  these  participants  said  that  they  had  not  applied  the 
training. 

The  difference  between  groups  in  this  case  falls  just  short  of 
statistical  significance.^  Considered  together,  however,  these  last  two 
analyses,  and  findings  for  Question  A. 4,  suggest  a trend  that  is  both 
consistent  and  in  line  with  managerial  assumptions.  Evidently  super- 
visor=subordinate  communication  does  make  a difference. 

One  may  well  asks  ”Are  communication  potentialities  being  sufficiently 
utilized  in  connection  with  training?”  Would  insistence  upon  more  and 
better  follow«=-up  by  supervisors  yield  a higher  return  on  the  training 
dollar? 

The  evidence  accumulated  so  far  does  seem  to  offer  considerable 
encouragement  to  the  belief  that  regular  application  of  procedures  like 
those  employed  in  this  project  could  contribute  substantially  to  that  end. 


^The  Fisher  test  yields  a one-tailed  P of  6% 
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C o FEEDBACK 


Question  Cols  How  effective  i s feedback  to  participants  and  supervisors? 

What  were  the  reactions  to  this  feedback  information? 

Follow-up  Question  4 for  both  supervisors  and  participants  was 
designed  to  check  on  readership  and  reaction  to  the  previous  reporto  It 
ran  as  followss 


a„  HAVE  YOU  SEEN  THE  PERSONNEL  RESEARCH  STAFF  REPORT,  "A  PATH  TO 
MANAGEMENT  DEVELOPMENT  AND  TO  THE  MEASUREMENT  OF  ITS  GROWTH"? 

bo  IF  YES,  DO  YOU  HAVE  ANY  COMMENTS  ON  THE  FINDINGS? 


The  proportion  of  YES  and  NO  responses  were  about  the  same  for  both 
groups  o 

Of  25  supervisors ? 

9 answered,  YES; 

16  answered,  NO, 

Of  37  participants s 
14  answered,  YES; 


23  answered,  NOo 

Nine  of  the  14  participants  who  had  seen  the  report  offered  comments 
Three  said  that  they  found  it  interesting  and  informat ive®  One  expressed 
surprise  at  the  "seemingly  large  number  of  participants  who  professed  not 
to  know  why  they  were  involved o"  Another  concluded  that  it  would  be  well 
to  devote,  more  time  and  thought  to  filling  out  questionnaires o 


One  participant  remarked  that  the  report  "raises  pertinent  questions 
This  participant  and  two  others  enlarged  upon  this  point  with  ideas  that 
deserve  to  be  quoted  in  fulls 

"The  findings  reveal  a need  for  better  communications 
between  the  supervisors  and  employeeo  The  findings  of 
this  survey  should  be  similar  or  applicable  to  other 
courses o Thus  if  the  supervisor  and  employee  are 
aware  of  the  deficiencies,  they  should  give  more  atten- 
tion to  them  in  future  courses  thereby  improving  the 
overall  training  effort," 
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”The  findings  point  out  the  faults  which  have  been  quite 
obvious  to  me  for  some  time,  i.e,,  lack  of  communication 
between  supervisor  and  employee,  and  determination  of 
employee*s  real  or  actual  training  needs.  Selection  of 
employees  to  attend  training  courses  (from  my  observation 
and  experience)  is  usually  done  haphazardly  without 
consideration  of  employee *s  needs  related  to  course 
content  plus  consideration  of  employee's  potential  and 
career  advancement.” 

”I  believe  that  the  report  has  made  crystal  clear  that 
there  is  an  urgent  need  for  people  responsible  for  train- 
ing programs  to  redouble  their  efforts  to  make  as  sure  as 
possible  that  the  right  man  gets  the  right  training  at 
the  right  time.  To  me,  this  suggests  career  counseling 
for  employees  and  supervisors  by  qualified  personnel 
officers.  If  done  on  a planned  basis,  this  could  supple- 
ment and  strengthen  career  development  plans  that  may  not 
be  adequate  without  such  counseling. 

”I  think  the  Department  should  be  commended  for  its 
current  efforts  in  the  career  development  field--there 
is  much  to  be  done.” 

This  last  participant  added  another  thought  on  a different  lines 

”0ne  point  that  I would  underline  is  the  fact  that  25  per- 
cent of  the  responses  indicate  that  the  participant  took 
part  in  the  decision  that  he  attend  the  course.  I think 
this  is  a symptom  that  bears  on  every  one  of  the  questions 
listed  on  pages  IV  and  V.  In  my  opinion,  the  initiative 
for  training  that  is  acquired  on  official  time  and  at  no 
expense  to  the  employee  should  rest  entirely  with  super- 
visors, training  staff,  and  top  management.  There  is 
plenty  of  room  for  employee  initiative  in  'extracurricular® 
work  in  the  USDA  Graduate  School  and  elsewhere.” 

Of  the  9 supervisors  who  had  read  the  report,  4 offered  comments. 

One  remarked  that  the  results  did  not  surprise  him.  Another  thought  that 
it  should  stimulate  better  selection  of  trainees  and  more  adequate  follow 
up.  A third  supervisor  felt  that  ”as  an  experimental  technique  the  study 
was  too  subjective  and  the  questions  were  too  susceptible  to  varying 
interpretations.”  Nevertheless,  he  did  acknowledge  that  the  study  "pointed 
up  some  of  the  deficiencies  in  the  existing  practices  in  evaluating  train- 
ing.” The  fourth  supervisor  expressed  surprise,  at  the  "relatively  small 
change  in  the  percentage  of  participants  who  were  able  to  suggest  specific 
applications  in  their  regular  jobs  before  and  after  training,”  He  explained 
that  he  had  taken  the  course  himself  and  had  found  many  opportunities  to 
use  it. 
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Question  Ce2;  Does  feedback  promote  follow-up? 

It  will  be  recalled  that  25  supervisors  reported  on  36  participants 
in  the  fol low-up. 

The  relationship  between  seeing  the  first  report  and  discussion  plus 
follow-up  on  one  or  more  participants  is  revealed  by  the  following  data. 

Of  9 supervisors  who  had  seen  the  first  report, 

7 reported  discussion  of  the  training  plus  specific  follow-up 
on  one  or  more  participants; 

2 did  not . 

Of  16  supervi sors  who  had  not  seen  the  first  report,  or  could  not 

recall  whether  or  not  they  had  seen  it 

4 reported  discussion  and  follow-up; 

12  did  not „ 

Since  the  difference  is  highly  significant  statistically,  the  answer 
to  our  question  is  self-evident.^ 


9 


The 


Fisher  test 


yielded  an  exact  probability  of 


1.67o, 
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D*  OTHER  COMMENTS 


Question  D.l:  What  other  reactions  are  expressed? 

Follow-up  Question  5 for  both  participants  and  supervisors  called 
for  Other  Comments » 

Such  comments  were  volunteered  by  12  participants  and  9 supervisors. 
The  comments  of  both  groups  covered  a fairly  wide  range. 

Several  of  the  participants  expressed  appreciation  of  the  value  of 
the  training,  although,  as  one  pointed  out,  it  is  not  a *’pink  pill”  nor 
a panacea.  Another  participant  said  that  he  had  recommended  the  course 
to  a colleague.  It  was  also  pointed  out  by  both  participants  and  super- 
visors that  one  may  use  parts  of  the  Kepner-Tregoe  approach  in  certain 
situations  without  using  the  entire  system.  Two  participants  remarked 
that  the  systematic  analysis  of  problems  tends  to  become  a subconscious 
part  of  one*s  reasoning  as  time  goes  on. 

One  participant  felt  that  a Kepner-Tregoe  type  of  course  which  was 
geared  to  government  problems  would  have  been  more  beneficial.  Another 
found  it  of  little  help  in  dealing  with  administrative  personnel.  Two 
more  mentioned  ^unique”  peculiarities  of  their  own  work  which  they 
believed  made  the  training  inapplicable. 

Among  the  supervisors,  two  emphasized  the  difficulty  of  identifying 
specific  benefits  directly  traceable  to  this  training.  One  supervisor, 
however,  said  that  the  course  had  made  his  participants  (he  reported  on 
several)  more  ^objective  and  helpful”  in  staff  meetings.  One  said  that 
he  was  recommending  another  man  for  the  training. 

Some  supervisors  made  unfavorable  comments.  According  to  one,  ”So 
far  as  thi s person  is  concerned,  I think  the  money  could  have  been  better 
spent.  We  wished  to  strengthen  his  ’Decision  Complex’  and  came  up  with 
’O’”,  Like  some  of  the  participants,  one  supervisor  felt  that  the  train- 
ing was  not  applicable  to  the  work  of  his  agency.  Another  said  that, 
after  sending  two  employees  to  Kepner-Tregoe  courses  and  three  to  a USDA 
Graduate  School  course  in  Federal  Executive  Development  Training,  he 
concluded  that  the  latter  was  more  beneficial. 

Finally  one  supervisor  doubted  the  value  of  the  questionnaire. 
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V.  DISCUSSION 


One  purpose  of  this  research  project  was  to  develop  and  try  out  a 
prototype  method  for  evaluating  managerial  training.  The  data  presented 
in  this  and  the  preceding  report  seem  to  show  that  it  is  possible  to 
obtain  reasonably  objective  and  practically  meaningful  evidence  concern- 
ing the  effectiveness  of  such  training. 

It  should  be  noted  that  the  evaluation  procedure  is  concerned  solely 
with  the  effectiveness  of  the  training  in  producing  observable  changes  in 
the  job  performance  of  the  participants.  Neither  participants  nor  their 
supervisors  were  asked  how  they  liked  the  course,  what  they  thought  of 
the  instructor,  how  good  the  presentations  were,  and  so  on.  Some,  indeed, 
volunteered  comments  on  such  matters,  but  these  comments  were  not  primary 
to  our  purposes. 

Our  object  was  to  find  out  what  the  participants  actually  did  with 
the  training  after  they  had  received  it,  how  they  used  it  in  their  work, 
and  whether  the  amount  and  kinds  of  communication  and  follow-up  that 
took  place  between  participants  and  their  supervisors  made  a difference 
in  the  participant  * s application  of  his  newly  acquired  learning. 

These  are  the  questions  one  must  ask  if  he  wishes  to  find  out  whether 
the  training  is  worth  the  cost.  Certainly,  it  is  legitimate  to  obtain 
information  about  other  points  as  well,  but  such  information  does  not  get 
at  the  heart  of  the  matter,  namely,  what  did  the  training  actually  accom- 
plish in  the  way  of  behavioral  change?  Specifically,  what  did  the  parti- 
cipant do  after  training  that  he  was  not  doing  before  training?  From  these 
two  reports  it  would  appear  that  this  kind  of  information  can  be  obtained, 
if  one  is  willing  to  take  the  trouble  to  get  it. 

The  findings  themselves  raise  a number  of  administrative  questions. 
Judgments  must  be  made  as  to  whether  these  situations  are  acceptable: 

Only  43%  of  the  participants  were  able  to  specify  in  detail  how  they 
had  applied  the  training  in  their  work; 

27%  were  unable  to  indicate  that  they  had  used  the  training  at  all; 

In  337,  of  the  cases  there  was  no  discussion  of  the  training  between 
participants  and  their  supervisors  during  the  eight  to  ten  months 
following  completion  of  the  course; 

In  447o  of  the  cases  no  definite  follow-up  efforts  were  reported  by 
the  supervisors; 

In  56%  of  the  cases  supervisors  were  unable  to  specify  what,  if  any, 
use  the  participants  had  made  of  the  training; 
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Feedback  on  the  first  phase  of  this  project  reached  only  367o  of  the 
supervisors  and  387,  of  the  participants. 

The  questions  raised  by  findings  like  these  become  more  insistent 
in  the  light  of  other  data  presented  in  this  report.  Thus,  the  evidence 
indicates  that; 

Asking  about  follow-up  plans  increases  the  likelihood  that  follow-up 
will  actually  occur; 

Feedback  promotes  follow-up; 

Discussion  and  follow-up  efforts  by  the  supervisor  increases  the 
chances  that  the  participant  will  find  applications  of  the  training 
in  his  work. 

If  then,  the  training  situation  seems  unsatisfactory  at  present, 
there  appear  to  be  ways  of  improving  it. 
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In  the  first  report  tnis  project  was  characterized  as  being  in  part 
an  action  research  prograTti.  This  means  that  the  research  effort  itself 
is  calculated  to  bring  about  a change  in  the  thing  that  is  being  inves- 
tigated--in  this  case,  irnproveTnent  in  the  effectiveness  of  training. 

But  to  achieve  any  real  irnproveirient  in  this  area  it  is  essential  not 
only  that  the  research  program  be  a continuing  one,  but  also  that  the 
findings  at  each  stage  be  communicated  to  all  and  pondered  by  all 
concerned. 

This  approach  to  evaluation  now  has  been  applied  to  a few  other 
training  programs.  Two  interagency  programs,  the  Seminars  in  Executive 
Development  (SED)  and  Seminars  in  Middle  Management  (SIMM)  were  mentioned 
in  our  first  report.  In  addition,  the  Forest  Ser'/ice  is  using  the  tech- 
nique in  a long  range  effort  to  evaluate  the  full-time  graduate  training 
received  by  some  of  its  personnel,  and  the  Agricultural  Marketing  Service 
has  employed  it  to  appraise  the  effectiveness  of  its  IMPACT  program. 

These  initial  developments  are  all  to  the  good.  But  are  the  evalu- 
ations still  being  carried  out  in  the  Kepner -Trego e courses  to  which  the 
Department  sends  participants?  Are  they  being  continued  systematically 
in  the  other  programs  just  mentioned?  Are  the  questionnaire  returns 
being  analyzed,  and  the  findings  communicated  to  all  interested  parties? 
Above  all,  what  is  being  done  toward  applying  the  information  to  minimize 
wasted  effort  and  to  obtain  better  managerial  training  in  the  USDA? 

VI.  RECOMMENDATION 

One  supervisor  doubted  the  value  of  the  questionnaire  used  in  this 
project.  His  point  is  well  taken.  If  the  procedure  described  in  these 
two  reports  is  used  as  a ’’one-shot”  affair,  then,  indeed,  it  has  little 
lasting  value.  To  quote  the  concluding  paragraph  of  our  first  report; 

”To  realize  the  potential  benefits  of  evaluation, 
however,  it  is  necessary  that  the  procedure  be 
carried  out  methodically  and  on  a continuing  basis. 

This  means,  among  other  things,  insisting  upon 
complete  returns  from  both  participants  and  their 
supe'jv/isors  on  all  questionnaires.  Unless  this 
requirement  is  observed,  the  entire  effort  becomes 
pointless.  If  management  itself  treats  the  evalu- 
ation casually,  others  will  do  so  too.  The  result 
can  be  an  altogether  unwarranted  complacency  regard- 
ing the  effectiveness  of  the  entire  training  and 
management  development  program.” 

It  is  recommended  that  the  method  descrioed  here,  or  appropriate 
variants  of  it,  be  applied  if  possible  to  all  management  development 
courses  on  a continuing  basis. 
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TRAINING  RESEARCH  QUESTIONNAIRE 
INSTRUCTIONS  FOR  SUPERVISORS 

It  has  now  been  six  months  or  more  since  your  employee  completed 
the  Kepner-Tregoe  course  in  ^'Management  Problem  Analysis  and  Decision 
Making,"  and  again  we  would  like  to  have  your  frank  reactions  on 
certain  points.  As  explained  in  the  questionnaires  which  were  filled 
out  earlier,  we  want  to  check  on  long-term  effects  of  the  training. 

Please  answer  the  questionnaire  for  each  employee  listed  below. 
Please  feel  free  to  react  favorably  or  unfavorably,  or  to  agree  or 
disagree  with  anyone  else's  point  of  view,  or  to  change  the  points  of 
view  you  may  have  expressed  last  time.  What  we  want  are  your  reactions, 
based  upon  your  own  experience  and  observations.  In  giving  your  answers, 
please  be  as  specific  as  possible. 

Two  blank  sheets  are  provided  for  your  responses  (do  not  hesitate 
to  use  additional  sheets  if  you  need  them).  Kindly  number  your  answers 
to  correspond  with  the  questions. 


Thank  you  once  more  for  your  time  and  trouble. 
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FOLLOW-UP  QUESTIONS  FOR  SUPERVISORS 


1.  Has  this  employee  discussed  the  Kepner-Tregoe  concepts  and  methods 
with  you  during  the  past  six  months  or  so? 

2.  a.  Have  you  been  able  to  follow  up  on  his  use  of  the  Kepner-Tregoe 

training? 

b.  If  yes , in  what  specific  ways? 

3o  a.  Have  there  been  any  situations  in  which  these  concepts  and 

methods  seem  to  have  been  helpful  to  this  employee  in  his  work? 
bo  If  yes , give  specific  examples. 

Co  If  n£,  what  do  you  consider  to  be  the  reasons  he  has  been  unable 
to  make  use  of  these  concepts  and  methods? 


4.  a.  Have  you  seen  the  Personnel  Research  Staff  report,  "A  Path  to 
Management  Development  and  to  the  Measurement  of  Its  Growth”? 
bo  If  yes , do  you  have  any  comments  on  the  findings? 


5o 


Other  comments? 


30 


TRAINING  RESEARCH  QUESTIONNAIRE 
INSTRUCTIONS  FOR  PARTICIPANTS 


It  has  now  been  six  months  or  more  since  you  completed  the  Kepner- 
Tregoe  course  in  ''Management  Problem  Analysis  and  Decision  Making," 
and  again  we  would  like  to  have  your  frank  reactions  on  certain  points. 
As  explained  in  the  questionnaires  which  you  filled  out  earlier,  we  want 
to  check  on  long-term  effects  of  the  training. 

Please  feel  free  to  react  favorably  or  unfavorably,  or  to  agree  or 
disagree  with  anyone  else's  point  of  view,  or  to  change  the  points  of 
view  that  you  expressed  last  time.  What  we  want  are  your  reactions 
based  upon  your  own  personal  experience.  In  answering  these  questions, 
please  be  as  specific  as  possible. 

Two  blank  sheets  are  provided  for  your  responses  (do  not  hesitate 
to  use  additional  sheets  if  you  need  them).  Kindly  number  your  answers 
to  correspond  with  the  questions. 

Thank  you  once  more  for  your  time  and  trouble. 
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FOLLOW-UP  QUESTIONS  FOR  PARTICIPANTS 


1,  a.  Have  there  been  any  situations  in  which  the  Kepner-Tregoe 
concepts  and  methods  were  helpful  to  you  in  your  work? 

b.  If  yes , give  specific  examples. 

2o  a.  Have  there  been  any  situations  in  which  you  tried  to  use  these 
concepts  and  methods,  but  they  did  not  help  you  much? 

b.  If  yes , what  do  you  consider  to  have  been  the  reasons?  Give 
specific  examples. 

c.  If  you  have  not  been  able  to  apply  the  training  at  all,  what  do 
you  consider  to  have  been  the  reasons? 


3.  a.  Have  you  made  use  of  these  concepts  and  methods  in  discussions 
with  your  supervisor  during  the  past  six  months  or  so? 
b.  If  yes , give  examples. 


4.  a.  Have  you  seen  the  Personnel  Research  Staff  report,  ”A  Path  to 
Management  Development  and  to  the  Measurement  of  Its  Growth"? 
b.  If  yes , do  you  have  any  comments  on  the  findings? 


5.  Other  comments? 
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Supervisors 


lo  HAS  THIS  EMPLOYEE  DISCUSSED  THE  KEPNER-TREGOE  CONCEPTS  AND  METHODS 
WITH  YOU  DURING  THE  PAST  SIX  MONTHS  OR  SO? 

Leve  1 ^ « *'Yes*' 

Level  2 - ”No'’  or  no  answer 

2,  a,  HAVE  YOU  BEEN  ABLE  TO  FOLLOW  UP  ON  HIS  USE  OF  THE  KEPNER-TREGOE 
TRAINING? 

b.  IF  YES  5 IN  WHAT  SPECIFIC  WAYS? 

Level  ^ - Response  gives  specific  details  or  examples. 

*'By  observing  the  greater  efficiency  and  effectiveness 
with  which  he  conducts  conferences  with  equipment 
development  center  directors  and  staff.  By  observing 
the  speedier  resolution  of  issues  relating  to  project 
selection  and  sequence.*’ 

**I  have  noted  on  certain  occasions  evidence  of  some  of 
the  things  which  he  learned  through  this  course.  For 
example.;)  his  care  and  attention  to  fact-gathering  and 
the  use  of  a blackboard  for  problem  analysis  prior  to 
the  decision  making  process.** 

Level  ^ - General  statement ^ no  specific  details  nor  examples, 

**I  have  kept  no  ’score  sheet’  on  his  use  of  this 
method  in  problem  solving  on  a case-by^case  basis. 

I do  know  from  discussing  particular  problems  with 
him  and  his  reference  to  ’K=T®  methods  he  is  alert  to 
his  training  received,** 

’’Indirect ly 5 in  organizing  for  workloads.” 

Level  0 - *'No**|  ’’Not  specifically.” 
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Participants 


2,  a,  HAVE  THERE  BEEN  ANY  SITUATIONS  IN  WHICH  YOU  TRIED  TO  USE  THESE 

CONCEPTS  AND  METHODS,  BUT  THEY  DID  NOT  HELP  YOU  MUCH? 

b.  IF  Y^,  WHAT  DO  YOU  CONSIDER  TO  HAVE  BEEN  THE  REASONS?  GIVE 
SPECIFIC  EXAMPLES. 

c.  IF  YOU  HAVE  NOT  BEEN  ABLE  TO  APPLY  THE  TRAINING  AT  ALL,  WHAT 
DO  YOU  CONSIDER  TO  HAVE  BEEN  THE  REASONS? 

Level  - Response  mentions  one  or  more  situations  or  types 
of  problems. 

Level  0 - Respondent  can  think  of  no  such  situation;  indicates 
that  the  concepts  and  methods  are  applicable  in  all 
situations;  or  has  not  tried  to  apply  them, 

3.  a.  HAVE  YOU  MADE  USE  OF  THESE  CONCEPTS  AND  METHODS  IN  DISCUSSIONS 

WITH  YOUR  SUPERVISOR  DURING  THE  PAST  SIX  MONTHS  OR  SO? 

b.  IF  YES,  GIVE  EXAMPLES. 

Level  ^ - Response  gives  specific  details  or  examples. 

"In  discussions  on  difficulties  concerning  procure- 
ment of  aircraft  services  from  commercial  sources," 

"In  discussions  of  how  best  to  set  up  a nationwide 
conference  to  achieve  desired  results--problem  included 
elements  of  politics,  race  relations,  illiteracy,  inter- 
agency relations." 

Level  1_  - General  response,  no  specific  details  or  examples. 

"Only  in  a general  way.  For  instance,  in  discussing 
a problem  I may  suggest  that  he  is  indicating  the 
cause  without  specifying  the  problem." 

"In  the  organization  of  reports,  in  expression  of 
considered  opinion." 

Level  0 - No;  only  in  a general  way;  not  specifically. 
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Participants  and  Supervisors 

, a»  HAVE  THERE  BEEN  ANY  SITUATIONS  IN  WHICH  THE  KEPNER-TREGOE 
CONCEPTS  AND  METHODS  WERE  HELPFUL  TO  YOU  IN  YOUR  WORK? 

b»  IF  Y^5  give  SPECIFIC  EXAMPLES  o 

3,  ac  HAVE  THERE  BEEN  ANY  SITUATIONS  IN  WHICH  THESE  CONCEPTS  AND 

METHODS  SEEM  TO  HAVE  BEEN  HELPFUL  TO  THIS  EMPLOYEE  IN  HIS  WORK? 

bo  IF  YES,  GIVE  SPECIFIC  EXAMPLES, 

Co  IF  WHAT  DO  YOU  CONSIDER  TO  BE  THE  REASONS  HE  HAS  BEEN  UNABLE 
TO  MAKE  USE  OF  THESE  CONCEPTS  AND  METHODS? 

Level  ^ »■  Response  mentions  specific  task(s)  or  problem(s), 

”YeSo  There  was  the  task  of  allocating  $1855000 
of  Accelerated  Public  Works  money.  The  problem  was 
to  use  the  money  for  highest  priority  jobs.  The 
Kepner»Tregoe  method  worked  well  for  this  job,*' 

"Together  we  made  a thorough  study  of  a potentially 
serious  personnel  situatioug  identified  the  problem j 
developed  and  considered  several  possible  solutions^ 
considered  the  consequences  of  each  possible  solu- 
tion and  selected  the  one  which  we  feel  will  eliminate 
the  problem  without  creating  adverse  consequences," 

Level  - Response  mentions  general  task  or  problem  area^  but 
gives  no  specific  examples, 

"Nearly  every  day  there  are  circumstances  where  I 
am  called  on  to  make  decisions  or  to  advise  my 
supervisors,  or  State  Office  program  specialists  on 
programs.  This  course  has  helped  me  to  analyze 
quickly  the  problems  as  presented 5 to  evaluate 
alternative  courses  of  action^  and  to  rseleet  and 
recommend  the  most  logical  solution," 

"A  decision  regarding  allocation  of  new  funds  was 
made  with  more  promptness  and  assurance  than  had 
b e en  ou  s t oma  ry , " 

Level  0 - No  situation  or  problem  cited|  no  use  of  Kepner-Tregoe 
methodology^  "Don^t  know?"  (Supervisors) 

"I  do  feel  that  the  Kepner-Tregoe  course  has  been 
of  some  help  to  me  in  my  work 3 although  it  is  very 
hard  to  be  specific  in  giving  examples," 
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’’Certainly  in  his  day-to-day  decision  making, ” 

Note : Only  four  supervisors  answered  3c.  Their  replies  are 
quoted  in  the  text  of  the  report. 

4.  a.  HAVE  YOU  SEEN  THE  PERSONNEL  RESEARCH  STAFF  REPORT,  ”A  PATH  TO 

MANAGEMENT  DEVELOPMENT  AND  TO  THE  MEASUREMENT  OF  ITS  GROWTH”? 

b.  IF  YES,  DO  YOU  HAVE  ANY  COMMENTS  ON  THE  FINDINGS? 

Level  - ”Yes,” 

Level  0 - ”No;”  ’’Haven’t  read  it;”  ’’Don’t  remember.” 

5.  OTHER  COMMENTS? 


Level  - If  any  comments  are  given 
Level  0 - If  no  comments  are  given. 
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Exhibit  5 shows  the  composition  by  agency  of  the  original  partici- 
pants and  those  who  returned  follow-up  questionnaires. 


Exhibit  5 

Distribution  by  Agency  of  Participants 
Who  Returned  Questionnaires 


Agency 

Original  Group 

No.  7o 

Follow- 

No. 

"Up  Group 

7o 

FS 

36 

68.0 

26 

70.3 

AMS 

5 

9.4 

3 

8.1 

ASCS 

5 

9.4 

4 

10.8 

ARS 

4 

7.5 

2 

5,4 

FCS 

1 

1.9 

1 

2.7 

NAL 

1 ' 

1.9 

1 

2.7 

scs 

1 

1.9 

0 

0,0 

Total 

53 

100.0 

37 

100,0 

It  is  easy  to  see  that  the  corresponding  percentages  agree  quite 
closely.  We  may  conclude  therefore  that,  as  far  as  agency  representation 
is  concerned,  the  two  groups  do  not  differ  significantly. 

Exhibit  6 shows  the  composition  of  the  two  groups  with  respect  to 
GS  level.  Again  the  differences  between  corresponding  percentages  are  too 
slight  to  approach  statistical  significance. 


Exhibit  6 

Distribution  by  GS  Level  of  Participants 
Who  Returned  Questionnaires 


GS 

Level 

Original  Group 

No.  7. 

Follow- 

No, 

-up  Group 

7. 

15 

1 

1.9 

0 

0,0 

14 

27 

51,0 

19 

51.3 

13 

16 

30.2 

10 

27.0 

12 

9 

16.9 

8 

21.7 

Total 

53 

100.0 

37 

100,0 
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Although  the  two  groups  evidently  do  not  differ  significantly  in 
composition,  it  is  possible  that  other  selective  factors  might  have 
operated  to  determine  who  would  and  who  would  not  return  questionnaires. 
Is  it  the  ’’superior”  participant,  the  one  who  ’’learned  more,”  who  tends 
to  return  questionnaires  like  ours?  Or,  is  there  a biasing  factor  work- 
ing in  the  opposite  direction,  so  that  the  participants  who  profited 
least  from  the  training  are  over-represented  in  the  returns? 

A partial  answer  to  questions  like  these  may  be  obtained  by  com- 
paring the  responses  of  ’’returnees”  and  ”non-retumees”  to  a few  items 
from  the  earlier  questionnaires. 

Prior  to  training,  the  participants  were  asked,  ’’What  do  you  think 
are  the  specific  objectives  of  the  course?”  (Question  Pre-l),  Exhibit  7 
shows  the  distribution  by  level  of  response  of  those  participants  who  did 
and  those  who  did  not  return  follow-up  questionnaires. 


Exhibit  7 


Actual  (A)  and 
Distributions 

Theoretically 
of  Responses  to 

Proportional 
' Question  Pre 

(P) 

-1 

Level  of 

Response 

Group 

0-1 

2 

Total 

A P 

A P 

’’Returnees” 

27  (27,2) 

10  (9.8) 

37 

”Non-Retumees” 

12  (11.8) 

4 (4.2) 

16 

Total 

39 

14 

53 

Each  cell  in  Exhibit  7 shows  the  actual  number  of  ’’returnees”  and 
’’non-retumees,”  and  (in  parentheses)  the  number  to  be  expected  if  the 
two  groups  were  represented  proportionately  at  each  level.  Obviously, 
the  actual  and  ’’expected”  counts  are  so  nearly  alike  that  a statistical 
test  of  significance  is  unnecessary. 
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Exhibit  8 shows  in  the  same  manner  the  distributions  of  responses  to 
Question  Pre-2:  ’*How  does  this  seem  to  fit  into  your  career  plans?” 


Exhibit  8 

Actual  (A)  and  Theoretically  Proportional  (P) 
Distributions  of  Responses  to  Question  Pre-2 


Group 

Level  of 
0-1 

Response 

2 

Total 

A P 

A 

P 

’’Returnees” 

18  (20.3) 

19 

(16.7) 

37 

’’Non-Returnees” 

11  ( 8.7) 

5 

( 7.3) 

16 

Total 

29 

24 

53 

Although  the  differences  between  actual  and  ’’expected”  frequencies 
are  somewhat  larger  than  in  Exhibit  7 5 they  nevertheless  do  not  approach 
statistical  signif icancco 

Question  Pre-4  asked  the  participant  whether  he  had  discussed  the 
objectives  of  the  course.,  its  relation  to  his  career  plans,  and  possible 
applications  of  the  training  with  his  supervisor.  The  response  distri- 
butions of  ’’returnees”  and  ”non=returnees”  are  compared  in  Exhibit  9, 
Once  more 5 the  differences  between  actual  and  ’’expected”  counts  are 
negligible. 


10 

Chi°square  equals  1,101 5, 
levels  of  confidence. 


significant  between  the  307,  and  207, 
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Exhibit  9 

Actual  (A)  and  Theoretically  Proportional  (P) 
Distributions  of  Responses  to  Question  Pre-4 


Group 

Level  of 
0-1 

Response 

2 

Total 

A P 

A 

P 

’’Returnees” 

20  (19.6) 

17 

(17.4) 

37 

’’Non -Returnees” 

8 ( 8.4) 

8 

( 7.6) 

16 

Total 

28 

25 

53 

Among  the  questions  asked  participants  immediately  following  train- 
ing was  the  following:  "Give  examples  of  situations  in  your  own  work  in 
which  the  concepts  and  methods  taught  in  this  course  would  be  applicable”'* 
(Question  Post  2=-a),  Forty-two  participants  answered  this  question;  of 
thesej  30  returned  follow-up  questionnaires  and  12  did  not.  As  Exhibit 
10  shows,  no  significant  differences  between  actual  and  ’’expected”  counts 
appeared. 


Exhibit  10 

Actual  (A)  and  Theoretically  Proportional  (P) 
Distributions  of  Responses  to  Question  Post-2a 


Group 

Level  of 
0-1 

Response 

2 

Total 

A P 

A P 

’’Returnees” 

16  (16.4) 

14  (13.6) 

30 

”Non-Retumees” 

7 ( 6.6) 

5 ( 5.4) 

12 

Total 


23 


19 


42 
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As  a final  check  on  comparability,  responses  of  the  two  groups  to 
Question  Post-=>4  were  examined.  This  question  asked,  "What  plans  do  you 
have  for  discussing  this  course  with  your  supervisor?"  The  results  are 
shown  in  Exhibit  11,  As  in  all  the  other  cases,  no  significant  differ- 
ences appeared  between  the  two  distributions. 


Exhibit  11 

Actual  (A)  and  Theoretically  Proportional  (P) 
Distributions  of  Responses  to  Question  Post-4" 


Group 

Level  of  Response 

0-1  2 

A P A P 

Total 

"Returnees" 

11  (12.9) 

19  (17.1) 

30 

"Non-Returnees" 

7 ( 5.1) 

5 ( 6,9) 

12 

Total 

18 

24 

42 

It  seems  then,  that  from  the  standpoint  both  of  composition  and  of 
"quality"-‘=the  latter,  as  shown  by  their  responses  to  the  two  earlier 
questionnaireS““the  follow-up  group  may  be  considered  a representative 
sample  of  the  original  participants.  This  permits  analysis  and  inter- 
pretation of  follow-up  responses  as  a meaningful  part  of  the  overall 
research  data.  In  other  words,  the  sample  seems  to  be  statistically 
adequate  to  satisfy  the  requirements  of  this  research. 
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